Extensions of Linear Independent Component Analysis: Neural and Information-Theoretic Methods

Thesis for the degree of Doctor of Technology to be presented with due permission for public examination and criticism in the Auditorium F1 of the Helsinki University of Abstract Obtaining information from measured data is a general problem which is encountered in numerous applications and elds of science. A goal of many data analysis methods is to transform the observed data into a representation which reveals the information contained in the data. Methods for obtaining such representations include principal component analysis , projection pursuit, cluster analysis, and neural unsupervised learning methods. In the eld of neural networks, it has been suggested that one important goal of sensory coding is redundancy reduction, which can also be seen as the goal of many unsupervised learning methods. An explicit method for representing data in a non-redundant way is linear independent component analysis (ICA), where the observed data vectors are linearly transformed so that the transformed vector has independent components. The basic form of the linear ICA is relatively well understood. In this thesis, some extensions of independent component analysis are studied. Nonlinear transformations are considered and some existing unsupervised techniques are interpreted as implementing nonlinear ICA. New com-putationally eecient algorithms for computing linear ICA are developed, and some existing neural methods are shown to have connections to information-theoretic contrast functions. A problem of overdetermined ICA basis is solved in the case of binary components and algorithms for computing the basis are given. Finally, statistical independence as a measure of redundancy is extended using algorithmic information theory. This extension leads to a new deenition of ICA, which includes the standard linear ICA as a special case.